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ABSTRACT 


Since ancient times, adversary modeling has been used 


during wargaming exercises in which military leaders have 





recreated past battles or simulated future battles in order 








to educate military professionals. Although the technology 








today is much different, adversary modeling still serves the 


same goals - to help military professionals learn tactics 





from past successes and mistakes. In the computer age, 
highly accurate models and simulations of the enemy can be 
created. However, including the effects of motivations, 


capabilities, and weaknesses of adversaries in current wars 





is still extremely difficult. 





Limit Texas Hold’em poker, with many attributes similar 








to real-world warfare, is an excellent test-bed to study and 





improve adversary modeling. For example, stochastic 


outcomes which deal with multiple independent agents, 





deception, and acting amidst uncertainty, are some of the 








aspects of poker that closely resemble important aspects of 


warfare. These attributes make poker a better choice as a 





study platform than other traditional games, such as chess, 


where there is no deception or uncertainty. 





The defined rules of poker provide researchers with a 


controlled environment to improve and test adversary- 




















modeling techniques. Perfecting adversary modeling in poker 
will allow simulators to improve and generate more accurate 
models for wargames, giving warfighters the advantage in 


current and future battles. 
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Qs INTRODUCTION 


A. HISTORY OF ADVERSARY MODELING 


The importance of adversary modeling has been known for 


centuries. Sun Tzu [1], the 6th Century B.C. military 








strategist wrote: 


If you know the enemy and know yourself, you need 
not fear the result of a hundred battles. If you 
know yourself but not the enemy, for each victory 
gained, you will also suffer a defeat. 


Adversary modeling has been used since ancient times in 
a military context during a process called wargaming. 
During a wargame, commanders seek to improve their battle 
plan by stepping through the plan with consideration given 
to the enemy’s actions, reactions, strengths and weaknesses. 
Adversary modeling is conducted by an intelligence officer 


who has studied th nemy’s capabilities and whose goal is 





to defeat the commander’s plan so as to improve the plan. 


Besides military applications, adversary modeling is 
used in a wide variety of areas. For example, in the 
computer-security realm, network-security professionals 
frequently create models of potential attackers in order to 
help them identify when their systems are being attacked. 
Additionally, adversary modeling has been studied and shown 
to improve bot performances in games such as Scrabble and 


RoShamBo [2],[3],[4]. 


3 Es Pre-Computer Adversary Modeling 


Games like Go and Chess wer used teach soldiers 








competence in battlefield situations. In these games, 





adversary modeling is not as important because they are 





perfect information games where all elements of the game 


(i.e., game board and game pieces) are known to all players. 





However, in actual wargaming situations, only limited 








information about the enemy is known and the rest must be 


inferred by an intelligence officer. Using the simplest 





adversary model, the intelligence officer acts as a friendly 





commander would act. While this approach does help find 





some weaknesses in a plan, it is far from being realistic. 





A much better model would simulat th nemy’s actions 


according to that enemy’s own doctrine. Although the 





benefits of this model are enormous becaus the enemy 
actions can reflect the leadership of a specific enemy 
commander, it necessitates a thorough understanding of the 
enemy commander’s tactics and observations obtained through 


vigorous analysis from many previous battles. 


2. Computational Approaches 


Since the advent of computers, wargaming has improved 





through more complex modeling and simulations. Using a 





computer and simulated battles, models of friendly and enemy 





units can fight with no loss of life, equipment, or other 


valuable resources. An accurate knowledge of an enemy’s 





doctrine, tactics, and motivations can tremendously improve 


the accuracy of these models and simulations. These 





modeling and simulation techniques have been incorporated 


into a commercial setting with the popularity of video 


2 


games. Today, countless video games simulate old battles or 





create fictional or fantastic scenarios allowing players to 





wage battles with different tactics. 





B. IMPORTANCE OF ADVERSARY MODELING 


In all of the situations described above, highly 
accurate models of opponents increase the utility of the 
game. In commercial computer games, this makes a more 
realistic and higher selling game. In the wargaming 


scenario, a better model of the enemy helps create a better 





plan to defeat the enemy. 


Military and Intelligence Community Adversary 
Modeling 


During the Cold War, adversary models were simpler than 


they are today because Soviet doctrine was relatively well 





known. Battles and wars could be simulated during the 





wargame based on knowledge gleaned from past battles, known 
tactics and commanders, and obvious motivations and morale 
of the soldiers. Since the end of the Cold war and the 


beginning of the War on Terror, adversary models have become 





increasingly difficult to create accurately. Not only do 








motivations of a terrorist differ greatly from the 











motivations of a soldier fighting for his state, motivations 





of different terrorist groups can be vastly different from 





each other as well. For these reasons, modeling in this new 


age of warfare is very difficult. 


2 Poker Adversary Modeling 





The game of poker provides an excellent test-bed for 
adversary modeling. Poker is a game containing stochastic 


c) 


events, imperfect information, multiple competing agents, 
and deception. Like the real-world scenario of warfare, 
adversary modeling substantially improves performance in a 


poker game. 


a. Introduction to Poker 


In our studies, we use Limit Texas Hold’em Poker. 





The game is played with blind bets that players must make 
before cards are dealt. The first person to the left of the 
dealer begins with a bet called the “small blind.” The 





person on their left follows the small blind with a bet 





called the “big blind,” which is twice the size of the small 
blind. These bets, similar to an ante, are used to 


instigate action, or encourage others to bet. All 





subsequent bets and raises in the first to rounds are the 


size of the big blind. 


A hand begins with each player being dealt two 
cards, called “hole cards,” only known to that player. The 


blinds are considered legal bets; therefore, the person to 





the left of the big blind is the first person to act after 





looking at their hole cards. This person now has three 
options - fold, call, or raise. A “fold” means that the 
player does not wish to continue and opts out of the hand. 


A “call” means that the player wishes to play for the number 








of bets that has already been established (in this case one 





- the big blind). A “raise” means that the player wishes to 


increase the number of bets from one (the big blind) to two 





(twice the amount of the big blind). This concept of the 








number of bets is sometimes referred to as “bets-to-go” or 





“bets-to-call.” Two bets-to-go simply means that all 





players who want to remain in the hand must pay two bets. 
4 


Play continues around the table until all players have 





either folded or called the highest raise. (Note: rules 
dictate that all betting rounds are capped at four bets.) 
If only one player remains, that player wins all the money 


in the pot and does not have to show their cards. The 





action up to this point is referred to as “pre-flop.” 


The “flop” is when three community cards (also 


called board cards) are placed face up in the center of the 








table. These cards are used by all players remaining in the 
hand. All remaining action is referred to as “post-flop.” 
At this point, another round of betting begins. The first 
player remaining in the hand to the left of the dealer acts 


first. He can “check” or “bet.” A check means that the 








player does not want to bet, and since no one else has bet, 
the player does not have to fold. A check keeps the game at 
zero bets-to-go while a bet makes it one bet-to-go. The 
betting continues as before, until everyone has folded or 
called the highest bet, or until only one player remains. 


Again the betting is capped at four bets-to-go. Now, a 


Ww 





fourth community card, called the “turn,” is dealt. This is 





followed by another betting round; however, all bets for 
this round and the final betting round are twice the size as 


the bets in the first two rounds. Finally, the “river” is 





the fifth and final community to be dealt. Following the 
river, there is a final betting round. At the end of this 


betting round, if more than one player remains, there is a 





“showdown” where the remaining players’ cards are revealed. 
The highest five-card poker hand-five cards can be taken 


from any combination of the player’s two hole cards and the 


five community cards-—wins the pot. The hand is now over, 
and the dealer position is moved one seat to the left to 


initiate a new hand. 


For simplicity, player’s actions can be viewed as 


three choices: raise, call or fold. Bets and raises can be 





abstracted together and called a raise. A bet is simply a 





special case of a raise when the betting round is zero bets- 
to-go. Similarly, a check and call can be abstracted to a 
call, the check being a special case of a call when a player 


does not want to increase the number of bets-to-go from 


zero. 
b. Importance of Adversary Modeling in Poker 
Adversary modeling is a vital part of maximizing 

your play in poker. Research has shown that the game- 


theoretic optimal solution does not necessarily result in 








the best poker player [5]. Game theory approaches result in 





good but defensive play, where a player will never lose big, 
but they will also never win big. A good model of a poker 


adversary will allow us to exploit their weaknesses, thereby 





allowing us to win larger amounts of money. 


Cc. MOTIVATION AND PURPOSE OF STUDY 


Poker allows us to improve adversary-modeling 
techniques in a structured domain. Not only does poker 
sufficiently limit the domain with its rule set, its 








stochastic elements and hidden information provide a high 





resemblance to real-world adversarial situations, providing 


an accurate test-bed for adversary-modeling research. 


In poker, every opponent has hidden information. More 
specifically, their hole cards are known only at the end of 
a hand, if at all. To apply this concept to warfare, it is 
evident that enemies have secrets. For example, the number 
of members in a terrorist cell is hidden and can change 
frequently, making that information impossible to know at 


all times. The dealing of cards is a stochastic event, 





which can be comparable to the numbers of disaffected youths 


that could be influenced by terrorist rhetoric. The 





strength of a player’s hand can be determined and compared 
to the other possibilities of an opponents hand based on the 
community cards. Correspondingly, the strengths of 
terrorist groups might be calculated and compared. The 
number of bets-to-call could parallel the cost of military 


“ 


or political actions. In poker, “pot odds” is a measure of 











the reward of an action compared to the cost of that action 


and could be analogous to many military operations. 
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II. RELATED WORK 


In the last decade, an increasing number of researchers 
began studying poker. For the last two years, a poker bot 
competition has been part of the annual Association for the 


Advancement of Artificial Intelligence (AAAT) convention. 





The fixed nature of this game (e.g. rules, betting actions) 





allows researches to build and improve adversary modeling 





techniques that can then b used in other domains. 
Adversary modeling is an important aspect of successful 


poker bots. 


A. THE UNIVERSITY OF ALBERTA’S COMPUTER POKER RESEARCH 
GROUP 


The University of Alberta’s (U of A) Computer Poker 
Research Group (CPRG) conducted the seminal research in this 
field. In [6], Billings provides a concise synopsis of the 
major accomplishment of the CPRG. Perhaps most importantly, 
they established a publicly available corpus of poker game 
data that can aid in adversary-modeling experiments. They 
studied limit Texas Hold’em—-recently focusing on heads-up 


games involving only two players. 


Their research began with poker bots that are derived 


from a rule-based system. As is typical in artificial 








intelligence, this method has only limited ffectiveness 





while the rules and knowledge base increase rapidly. The 





CPRG then attempted to calculate optimal play game 





theoretically. Finally, the CPRG experimented with using 





game-tree search methods to make decisions that result in 





the highest expected value. Varying degrees of adversary 


modeling are attempted by the CPRG, as discussed below. 


4 Eo Knowledge-Based Poker Player 


The first iterations of the U of A’s CPRG’s poker bots 











used knowledge-based artificial intelligence to establish a 
baseline. Only average poker play was attainable before th 
knowledge base and rules became too large and complex. The 








adversary modeling performed in this poker bot was based on 
observed statistics. The crucial information to deduce is 
the adversary’s hole cards. In the CPRG’s studies, the 
opponent’s hole cards are abstracted into 169 distinct 
hands. There are 13 different ranks, Two through Ace, and 


the cards are either suited or unsuited-making 169 distinct 





hands. 


The simplest starting point for the probability of an 
adversary’s hole cards is to assume a flat probability 
distribution function. This will provide a baseline, but 
will not correctly represent the probability of an adversary 
playing those hands because most players will play “better” 
hands with more probability than “worse” hands. The key 
variable is to determine which cards an opponent deems 


“better.” 


Using the “reasonable man” approach, the CPRG developed 





a generic adversary model (GOM) to infer which hole cards an 
average player is going to play. Billings et al. calculate 
an income rate, which is the expected value, for each 


possible pair of hole cards using simulations in [8]. 





Obviously, a “reasonable man” is less likely to play hands 





that result in a negative income. They assign probabilities 
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to each of the 169 starting hands that are based on the 
calculated income rate of that hand. As the play of a hand 
unfolds, they adjust these probabilities based on actions in 


a hand. For example, if the adversary raises, the 








probabilities assigned to the hands with high income rates 





are increased, while the probabilities for the hands with 








low income rates ar decreased. The increases are done 
based on rules that are applied to all players. However, 
not all players act as this GOM does. Some players are 











attracted to straights and flushes and are thus more likely 
to play cards that have a better chance of making those 


hands. 


The CPRG performs specific opponent modeling (SOM) by 
changing the weights differently for each individual 


adversary. For example, if an adversary usually bets with a 





flush draw, their algorithm will increase the probabilities 


of those hands that give the adversary a flush draw. In 





order to deduce the probabilities to use at the start of a 
hand for a specific adversary, the CPRG maintains counts of 
betting frequencies in certain contexts of the game. As 
discussed in the introduction to poker, there are three 


actions: bet, call or fold. Their system tracks the 





frequencies of these actions in twelve different contexts: 
based on the betting round (pre-flop, flop, turn, river) and 
the number of bets-to-call (zero, one and two or more). 


Over time, these frequencies would begin to evolve and could 





lead one to make assumptions about an adversary. For 





example, if a player bet 35% of the time after the flop when 


there are zero bets-to-call, one could assume that the 











adversary would bet with the top 35% of hands, or the top 


30% of hands and the other 5% based on strong drawing hands. 
ail 





For pre-flop frequencies, these percentages are mapped back 


to the income rates. Post-flop, the frequencies are mapped 








to a hand strength based on possible adversary hole cards 


combined with the board cards. In [8], the CPRG admits that 








this method is flawed becaus it is based on the CPRG’s 


calculations of income rates and hand strengths, which may 








be different from how the adversary calculates the strength 


of their hand. 


In [9], the CPRG improved this method of adversary 





modeling based on the results of experiments with Artificial 
Neural Networks (ANNs). They used 19 different aspects of 
the game context as inputs to the ANN which would then 


produce a likelihood of a raise, call, or fold from an 








adversary. They determined that ANNs were good at filtering 


out noisy aspects of game contexts, but required too many 





historical hands before becoming accurate. Thus, ANNs are 
not feasible for the real-time nature of poker. However, 
they did ascertain that “last bets-to-call” and “last 
action” were important factors for an adversary’s decision. 
These two dimensions of the game were added to the 
statistical model described above which produced improved 


results. 


In the methods described above, there is minimal use of 
the board cards in the context of the game, which seems to 


be a conspicuous weakness. 


2. Game Theoretic Methods 


The CPRG devotes time to finding the game-theoretic 


optimal solution at each decision node. They apply a 








randomized mixed strategy to the adversary’s actions. With 
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no adversary modeling done in thes xperiments, the actions 
of the poker bot are only based only on known cards. The 


play of their bot improves significantly over the knowledge- 








based system and is even able to initially play well against 











a professional poker player. However, given more time, th 
professional is able to discover weaknesses and can exploit 
the bot [5]. 

3. Game Tree Search Methods 


In their next set of experiments, the CPRG employs 





methods that search game trees in order to maximize the 





expected value (EV) of their decisions [10],[11]. In their 











game tree, there are four different types of nodes: chance 
nodes, adversary decision nodes, program decision nodes and 
leaf nodes. The chance nodes simply relate to the possible 


cards that could follow based on the known cards up to that 





point. The program decision nodes are where the program 
decides which action will result in the highest EV, with 


some variability added to disguise the program’s play. The 








adversary decision nodes are an estimated probability that 


the adversary will take each action: raise, call, or fold. 





This probability is based on counts of past actions at the 
corresponding point in the game tree and is in no way 
affected by the cards the adversary holds or the community 
cards, even if the previous counts ended in a _ showdown, 


where the adversary’s cards are revealed. The leaf nodes 





contain the EV of that node and the probability of winning 
the pot. The probability of winning the pot is determined 





using a histogram of previous hand strengths that the 


adversary has shown at showdowns that correspond to that 





leaf in the game tree. The program will compare its hand 
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strength at that leaf to the hand strength histogram of the 





adversary to determine the probability of winning the hand. 





This method uses abstractions when the game tree is 


incomplete in order to b ffective when little information 





is known. One abstraction is obtained by using all branches 
of the game tree that have the same number of bets and 


raises, ignoring when the bets and raises are made. 





Another, finer-grained version of that abstraction uses all 


branches with the same ordered pair of the total bets and 





raises of both players. A more coarse-grained abstraction 
is simply the total number of bets and raises by both 
players. Another form of abstraction considers only the 


final size of the pot. In their experiments, the CPRG uses 





a combination of all of these abstractions. The 


abstractions are weighted stronger for the finer granularity 





of the abstraction and a mixture of all is used based on the 
weighting system. Generic adversary models are used as 
defaults until enough hands are recorded to make the 


specific adversary modeling precise. 


This method completely ignores the fact that the board 
cards will factor into the adversary’s decision making 
process. Additionally, a high computation time is needed 


for all decisions becaus th nae gam tr must be 





searched to completion for each decision. 








4. Bayes’ Bluff 

In [12], Southey, et al, experiment with a 
probabilistic model for opponent modeling. Each player has 
a strategy that is known only by them. Each player also has 


an information set for each hand consisting of the cards 
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visible to them. 


Using Bayes’ 


Rule, 


the probabilities of an 


opponent playing different strategies are calculated using 


the observat 


and hands 





that 


are 


folded. 


Next, 


the 


authors 


use 


tions of all hands—hands that go to a showdown 


the 





posterior distribution over the strategies to determin 








best 


response is the action that results in the highest 


value. 


response to an opponent in the current hand. 





th 


The best 





xpected 


The authors tested this method against various other 


poker bots. 





The results show that this model is effective 


in countering an opponent’s 


strategy in as 


little as 200 


hands. 


B. OTHER RESEARCH 


As poker increases in popularity revealing more 


complexities, other researchers have in with 





joined 


The most influential 





experiments of their own. methods for 


the research described in this thesis follow. 





1. Carnegie-Mellon University Method 


In [13],[14],[15], Gilpin and Sandholm describe a 





method of calculating th 





game theory equilibrium and then 


use Bayes rule for predicting the hole cards of an opponent. 





Offline, they compute optimal strategies for playing the 


automated 


the 


pre-flop and flop rounds. They first use 





abstraction techniques to condense the complexities of 


game. Then, they perform equilibrium computations using 








linear programming to calculate the expected value of future 


stochastic events (cards dealt in the upcoming turn and 


river rounds) without regards to future bets. During the 


turn and river rounds, the authors apply Bayes’ rule to 


calculate the probability of all possible hole cards based 
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on the computed strategies and the observed actions in the 
prior rounds. This method is computationally expensive but 
accounts for game context more than many other methods 


described in this thesis. However, the authors do not use 








any information from previous hands to influence action of 
the bot. Although their poker bot did win small amounts of 


money in their early experiments, the authors could not show 








that their poker player preformed better than the expected 





variance of Texas Hold’em [13]. Later results in [14], [15] 





show that their improvements produced ae statistically 


Significant win rate. 


2. Bayesian Networks 


There hav been several researchers who conducted 





experiments using Bayesian networks in [16],[17],[18],[19]. 





Although Korb, et al, and Boulton [17], [18] describe 
research conducted using another form of poker (Five Card 
Stud), it is useful to discuss their use of Bayesian 
networks which is the basis for later models that Carlton 


describes in [19]. 


In [20], Russell and Novrig describe a Bayesian network 
as a directed acyclical graph in which each node represents 
a random variable and each arc represents influence of one 
node on another node. Conditional probability tables are 


used to quantify the effect that parent nodes have on the 








child. The biggest drawback of using Bayesian networks for 
modeling opponents is the need of thes defined 
dependencies. The authors of [16] use dependencies among 





such game attributes as position, action, pot odd, hand 


strength, etc. However, not every poker player uses the 








same variables nor is everybody’s dependencies the same as 
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the authors’. This is evidenced by fact that the Bayesian 





networks shown in [17], [18],[19] use different nodes and 





arcs in their models. 


In [19], Carlton creates a generic opponent model by 


using self-play to initialize the conditional probability 





tables. This bootstraps the Bayesian network in order to be 








more ffective at the start of play against an unknown 
opponent. Then, a generic opponent model is created by 
editing the conditional probability tables according to the 


actions of a specific opponent during game play. 


The authors of these papers show little accuracy in 





their results. Carlton showed the best results in [19], but 
was still not able to beat human opponents or the state-of- 


the-art poker bots. These authors suggest that a more 








complex Bayesian network or a dynamic Bayesian network may 





yield better results. Dynamic Bayesian networks allow the 
relationships between the nodes to change at different 
stages of the game, but the dependencies still need to be 
defined. 





Cx RESEARCH CONDUCTED IN THIS THESIS 


1. The Use of Game Context 


Most of the methods described above made little use of 





the context of the game. In poker, this would be the 
community cards and the actions taken given these community 
cards. Additionally, the cards revealed at showdown can be 


rolled back to give insight into the decision made earlier 





in the hand. 
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The methods that do use game context use Bayesian 
Networks where the variables and dependencies are hard- 
coded. This, as discussed above, does not work well against 
opponents who do not use the same variables and 


dependencies. 


2. Hidden Markov Models 





Hidden Markov Models (HMMs) have an advantage over th 
methods describe above. Using HMMs, one can take into 
account the entire context of the game without defining the 


variables and dependencies that an opponent might use to 





make decisions. The hidden states in the HMM can represent 





the variables and dependencies used by an opponent to make 
his decisions. Furthermore, training the HMM for different 
opponents over different sequences of actions during the 
hands of a game allow the HMM to accurately represent 


different opponents. 
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III.DATA GATHERING AND DESIGN OF EXPERIMENTS 


A. DATA GATHERING 


ua University of Alberta’s Corpus 


The University of Alberta collected data from IRC-based 


poker rooms for years. This data is available online [21]. 





This corpus is used for much of the research conducted by 
the University of Alberta and other scientists. The corpus 
consists of a separate folder for each month of play. 
Within each month folder there is a hand database file, a 


hand roster file, and a player database folder. 


The hand database file lists, from left to right, a 
timestamp for the hand, the position of the dealer, the hand 
number, the number of players dealt in the hand, the number 
of players, the amount of money in the pot at the flop, 
turn, river, and showdown, and the community cards that were 


dealt (See Figure 1). 














797211363 1 LOT Sh Bk 9S 3/125 3/185 1/205 2c 9s ?e 9d Js 

797211456 al. 198 9 2/45 o/o o/0 1/55 Jd Qs 6s 

797211529 1 199 9 3/65 2/85 0/0 1/105 Qh 2c Se 6s 

797211616 1 200 8 4/80 3/140 3/200 2/280 Ts 4d 4c 7s Jc 

79211721) <a. 201. 8: 47 80 4/120 2/160 2/200 7d 6h 5d 9c Te 
Figure 1. Example hand database information. 


The hand roster, shown in Figure 2, consists of the 





timestamp for each hand, the number of players dealt in that 


hand and the user name of each player dealt in that hand. 
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797210666 9 Quick Winner??? derek greg gunner jims johnr sagerbot shinner 
797210948 68 Winner??? derek greg gunner jims johnr sagerbot shinner 
797211062 8 Winner??? deadhead derek greg gunner jims sagerbot shinner 
797211160 7 deadhead derek greg gunner jims sagerbot shinner 
797211251 7 deadhead derek greg gunner jims sagerbot shinner 
797211363 7 deadhead derek greg jims k*man sagerbot shinner 

Figure 2. Example hand roster information. 

The player database folder contains a separate file for 


each player who played at least one hand during 





that month. 


These files list the following information for each hand in 


which the player participated 








amount they won from the 


the timestamp of the hand, 
that hand, 
their actions, 


of the hand, 


(See Figure 3): their name, 
the number of players dealt in 
their position relative to the “dealer” position, 
the amount of money they had at the beginning 
the amount they contributed to the pot, the 


pot, if any, and their hole cards, 


if they were involved in a showdown. 





























sagerbot 797210868 93 1 Bk b b ke 1740 60 0 Od Jc 
sagerbot 797210948 8 1 Bc kf = = 1680 20 0 
sagerbot 797211251 7 Vc re b b 1660 80 205 9s 8s 
Sagerbot: 7972281721" .8: 2° Bc oc b b 1785 70 200 8h Ah 
sagerbot 797211886 8 8c c f = 1910 30 0 
sagerbot 797212372 6 2 Br b b = 1880 60 140 
sagerbot 797213334 10 2 Bk b = = 1890 20 55 
sagerbot 797213396 10 1 Bc br br 1925: 130) 330° Jd. 7d 
Figure 3. Example player database information. 
All information needed for this research was 


ascertained using the above files. 


In addition to the corpus of data, 


the University of 





Alberta provides basic, poker related code [22]. They have 
java source code files for a card, a deck, a hand, and a 
hand evaluator. The first three are simple classes to 
represent important concepts in the game. The hand 
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evaluator assigns an integer to every possible five-card 
hand such that a higher hand will be assigned a larger 


integer and two equal hands will be assigned the same 








integer. This class returns the integer representing the 





strength of the hand for any input of cards numbering 


between three and seven. 


2. Creating Hand Histories from Corpus 


Perl code was used to create hand histories for players 
with the most hands, which is based on the size of the 


player’s file in the player database. Chosen at random, 








data from May, 1995 was used in thes xperiments. The hand 





histories are files that contain all the information about 


the actions of all the players in each hand in which the 





target player participated. This data was mined from all 











the other player database files in the given month. 


3. Composition of the Action Vector 


For this research, an action vector was created for 
each action performed by the target player (See Figure 4). 


The action (ACT) was limited to raise, call, or fold, based 





on arguments described in the explanation of poker in 
Chapter I. The following information about the board cards 
was used: board score (BS), probability of a straight draw 
(PSD), the probability of a flush draw (PFD), the 
probability of a straight (PS), the probability of a flush 
(PF), and the Boolean concerning if the board contains a 
face card (FC). This data is set at zero for all actions 


that occur pre-flop. The board score is an integer returned 





from the University of Alberta’s hand evaluator class that 





represents the strength of the board cards alone. 
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When a poker player has a potential to make a good hand 
but needs another card, the player is said to be ona 
“draw,” (e.g. four cards of the same suit is called a flush 


draw). Flushes, straights, and draws to straights and 





flushes were modeled using probabilities. To obtain a 
probability of having a flush or a straight, every possible 


two-card combination of the remaining cards that when added 





to the current board cards makes a straight or a flush is 
divided by the number of all possible two card combinations 


to obtain a probability. A similar method is used to 








determine the probability of a draw, except a third card is 





added to represent the next board card to be dealt. 


In addition to the board information, the following 


information is tracked for every action: the number of 





players still in the hand who act before the target player 

















(PA), the number of people who act after the target player 





(PB), the number of bets-to-call (BTC), the pot odds (PO), 
and the amount of money the player has when he performs each 
action (POT). “Pot odds” is a term that represents a 
player’s reward-to-risk ratio and is the quotient of the 
amount of money already in the pot and the amount to call 


the current bet. 


The final information in the action vector is only 





available when the target player reveals their cards at a 
showdown. These showdown cards are used for all actions 


that the player conducted in that hand to determine the 





strength of the players hand relative to all possibilities 





(HS). For pre-flop strength, a lookup table was used that 





contains probabilities of having the best two-card hand. 





This probability is based on research by Sklansky [23], a 
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professional poker player, and Billings [6]. After the 
flop, the hand evaluator class discussed above is used along 


with the method similar to the one used to determin th 





possibility of a straight or flush. Every possible two-card 


combination is added to the board cards. The number of 





combinations that return a higher integer than the player’s 
hand is divided by the total possible combinations to obtain 


a number between one and zero. This number is used to 





represent the strength of the player’s hand. 

















BS PSD PFD PS PF Pi PB BTC PO HS POT Fc ACT 
oO 0.0 0.0 0.0 0.0 z 0 1 0.0 0.650 1740 FE B 
1995 0.0683 0.0895 0.0 0.0 o 1 ia) 0.0 0.8501 1740 T B 
25980 0.7487 0.0953 0.2836 0.0 ia) 1 0 0.0 0.9391 1730 F Cc 
337772 O.7782 0.1017 0.2960 0.0 ia) 1 0 0.0 0.9391 1720 a F 
337772 0.7782 0.1017 0.2960 0.0 bi 0 1 0.1666 0.9391 1720 F R 
18) 0.0 0.0 0.0 0.0 2 1 2 0.1538 0.0 1680 F Cc 
2135 0.0683 0.0895 0.0 0.0 ia) 2 0 0.0 0.0 1660 T F 
2135 0.0683 0.0895 0.0 0.0 2 0 1 o.1 0.0 1660 T F 
ia) 0.0 0.0 0.0 0.0 1 2 2 0.3076 0.0 1660 F B 
ia) 0.0 0.0 0.0 0.0 1 1 iL 0.2857 0.0 1660 F Cc 
Figure 4. Example action vectors 
4. Data Mining Hand Histories for Information 








Java code was written to step through the hand 
histories to make the action vectors described above. All 


the vectors for a given hand are stored in one file. These 





files are labeled with a number and the strength of the hand 


at the river. The strength of hand is defined as high, 
medium, low, and folds. Folds are hands that were folded 
and the hole cards remain unknown. For the remaining 


categories, the hand strength, as described in the previous 
section, is used. High is defined as 0.70 and higher. 
Medium is defined as greater than or equal to 0.40, but less 


than 0.70. Any hand lower than 0.40 is defined as low. An 
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additional file containing every vector is created and is 





used to determine clusters of hands for use in the following 


experiments. 


B. DESIGN OF EXPERIMENTS 


1. Hidden Markov Models 


A Hidden Markov Model (HMM) is a statistical model used 
to describe the state of a changing environment [20]. The 


states represent different values of discrete random 





variables over time. If one assumes a Markov process, a 
process in which the current state only depends on the 
previous state and not earlier states!, an HMM is useful 


when there is noise or uncertainty in the environment. In 





an HMM, the states are hidden or unknown but determine th 


observabl vidence emitted by the model. 





a. Structure of the HMM 


An HMM consists of a set of states, a start 





distribution, a transition matrix, and an observation 





matrix. The states are used to represent the hidden (or 
unknown) variables in a random process. The start 


distribution shows the probability of beginning in each 





state. The transition matrix contains the probability of 
moving from one state to any other state in the model. An 


HMM may allow only one path through the model, a linear 





model with no jump-ahead, or it may be possible to go from 


any state to any other state, an ergodic model, or some 


1 This describes a first order Markov process, in a second order 
Markov process, the current state only depends on the previous two 
states, and likewise for third and fourth order processes. 
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variation in between these two models. The observation 
matrix describes the probability of seeing a given 


observation in a particular state. 


There are thr tasks normally associated with an 





HMM: 


e Evaluation: given the parameters of the model, 
compute the probability of a given observed 
sequence using the forward-backward algorithm. 


e Decoding: given the parameters of the model, 
compute the sequence of states that most likely 
generated the observed sequence using the Viterbi 
algorithm. 




















° Learning: given an observed sequence or set of 
sequences, calculate the model that best explains 
the observation sequences using the Baum-Welch 











algorithm. 
b. Training and Testing 
For the purposes of th xperiments in this 





thesis, it is not necessary to comput the sequenc of 





states that generate the observations. In abstract terms, 





the states of the HMM are supposed to model what the player 





believes about the strength of his hand. The observations 
are his actions (raise, call or fold) and the game context 
at the time of his actions. The Baum-Welch algorithm is 


used to train the HMMs used in these experiments. Once the 





HMMs are trained, the forward-backward algorithm is used to 
determine which HMM was mostly likely to produce a given 


sequence. 
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2. Using Hidden Markov Models 


Experiments with HMMs were conducted in Matlab. For k- 





means clustering, fast k-means code for Matlab was used 
[24]. HMM Toolbox for Matlab is used for all of the HMM 


operations [25]. 


a. Vector Quantization of Game Context 


K-means is an algorithm for grouping large amounts 





of data into k different groups. The objective is to 





minimize the total distance from every data point to one of 


the centroids. To accomplish this task, k centroids are 





chosen throughout the space at random. Then, each data 


point is assigned to the closest centroid, creating k 





clusters of data. Next, ignoring the current centroids, 
centroids for the k groups are re-calculated and placed at 
the center of each of the k clusters. Again, each data 
point is assigned to the closest centroid. The algorithm 
repeats a given number of times or until the distance 


between successiv centroids is below some minimum 





threshold. Fach of the k centroids is labeled with an 





integer, 1 through k. The algorithm returns the integer, k 
representing the centroid closest to each of the data 


points. 


For thes xperiments, k-means was used to reduce 











the number of different sequences used to train the HMMs. 


This is similar to assuming that hands would be played 








Similarly during similar situation in a poker game. The 





following numbers of centroids were used in the experiments 


in this thesis: 50, 75, 100, 175, 250, and 500. Two 





dimensions of the action vector are eliminated befor th 
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clustering process: 1) the Boolean variable for face card 
present (FC), and 2) the action (ACT) - raise, call, or 


Odds. The k-means algorithm returns the 11 dimension 





cluster centroids and an integer (1 through k) representing 





that centroid. For simplicity, the integer representing the 





centroid is used in the experiments instead of the vector. 





In order to retain the information for FC and ACT that was 


not used in clustering, digits are appended to the end of 





the integer representing the cluster center. First, one 
digit is appended to represent FC - a “0” for false and a 


“1” for true. Finally, the second digit appended represents 





the action - the label “0” means fold, “1” stands for call, 


and “2” represents raise. At this point, each action vector 





is represented by one integer. For example, th xperiments 


with 50 centroids uses integers ranging from 100 to 5013; 





for experiments with 250 centroids, these integers range 


from 100 to 25013. 


Bb. Representing a Hand for Training and Testing 
HMMs 


In order to train the HMM, the input training 
sequences must contain all the actions of one hand on a 


Single line. Furthermore, each hand must be of equal 








length; therefore, ach hand is padded with integers to 
ensure that each sequence is of equal length. Since zero 
cannot be used as an input, an integer higher than any 
possible value of an action vector is used - 5014 for the 
50-centriod experiment and 25014 for the 250-centroid 
experiments are examples. Any hand in which the player’s 
first action was a fold was not used for training or 


testing. Figure 5 shows ten example hands from the 100- 
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centroid HMM. Notice that all hands end with several 





instances of padded integer - 10014 in this case. In the 
first hand in Figure 5, the first action vector is 
represented by 2601. 26 is the label of the vector 
quantized game context, the value of the Boolean FC is 0 and 





the action (ACT) is a call, represented by a 1. The second 
action of the hand is represented by the 2612: 26 for the 
game context, 1 for the presence of a face card, and 2 for 


the action of a raise. 





2601 2612 8412 4611 4611 10014 10014 10014 10014 10014 10014 10014 
2601 8202 8202 8202 86211 68211 10014 %10014 10014 10014 10014 10014 
2602 2601 8202 8202 8202 10014 10014 10014 10014 10014 10014 10014 
9602 9601 9601 3701 68711 %10014 10014 10014 10014 10014 10014 10014 
9601 9612 9611 4111 4111 8912 10014 10014 10014 10014 10014 10014 
9602 9601 3701 5901 10014 10014 10014 10014 10014 10014 10014 10014 
2601 2612 2611 8111 86111 5211 10014 %10014 10014 10014 10014 10014 
2601 2611 6411 6811 10014 10014 10014 10014 10014 10014 10014 10014 
4801 4812 9312 9912 10014 10014 10014 10014 10014 10014 10014 10014 
2602 2601 3812 3811 412 10014 10014 10014 10014 10014 10014 10014 























Figure 5. Example training and testing data. 


c. Experiments with Four HMMs 


The first experiment is to determine if HMMs are 


capable of categorizing a hand as a high, medium, low, or 





fold hand. To accomplish this, eight files are created for 





the player, two for each category of hands: high, medium, 
low, and fold hands. Eighty percent of the hands are placed 


in training files and twenty percent are placed in testing 





files. The HMMs used during these experiments have either 


four or eight states. The models used were rgodic; 





transitions are allowed from every state to any other state. 


Four HMMs were trained, one corresponding to each category 
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of hand (high, med, low, and fold) using the files 
containing eighty percent of the hands. The held-out twenty 
percent are then used to test this process. For observation 


sequences, the first action of a hand is used, then the 








first two actions are used, and so on, until the entire hand 
is used for a sequence. At each point, the forward-backward 
algorithm was used for each of the four HMMs in order to 
determine which HMM was mostly likely to produce the 


sequence so far. 


d. Experiments with Three HMMs 


A second = set of experiments was conducted 
Similarly to the method above. The only difference was that 


no fold data was used. Therefore, only three HMMs were 





trained. The HMMs were used to attempt to determine a high, 


medium, or low hand. 


e. Experiments with Two HMMs 


In the third set of experiments, a different 
method was used. Instead of one HMM per category, only two 


HMMs were used for each experiment. These xperiments 








attempt to classify hands as fold or not-fold, high or not- 


high, medium or not-medium, and low or not-low. As an 





example, in the fold or not-fold experiment, all of the 





high, medium, and low data was put into one file and used to 
train one HMM instead of three different HMMs, mutatis 
mutandis for high, medium, and low experiments. Again, the 


data was separated into eighty percent training data and 





twenty percent held-out testing data. Again, the forward- 
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backward algorithm is used on each sequence of the testing 
data to determine which of the two HMMs most likely produced 


the sequence. 
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IV. RESULTS AND ANALYSIS 


A. RESULTS AND ANALYSIS 


Accuracy, precision, recall, F-score and baseline F- 


score were all used to evaluate the performance of the HMMs. 





Accuracy is the number of predictions correct divided by the 


total number of predictions. Precision is the proportion of 





the predictions of X that were correctly labeled-X being the 
possible categories of high, medium, low, or fold hands. 
Recall measures the proportion of X’s in the corpus that 
were correctly labeled X. The F-score is the harmonic mean 
of recall and precision given by the following formula, 


where F is the F-score, P is the precision, and R is the 








recall: 
2 
F= 
—+— 
The F-score is used to balance th recall and 
precision. In order to attain a high F-score, both the 


recall and precision must be high; therefore, one cannot 


improve one measure at the expense of the other measur 








The baseline F-score is calculated using the F-score formula 





as if every prediction was xX. Therefore, the recall will 





always equal one and the precision will be proportional to 





the frequency of X. This is used too measure whether or not 








the performance of the HMM is better than chance. The 
baseline F-score is referred to as baseline for the 
remainder of this thesis. If the F-score is higher than the 
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baseline, the HMM can predict better than chance and 








assuredly the data contains information that can be used for 


prediction. 


The highest accuracy of the HMMs in this thesis was 


around 85%; however, most HMMs only attained 60% accuracy. 








Although the accuracy is not consistently high, many scores 








were Significantly above the baseline score. Additionally, 
a high precision when predicting fold hands and high hands - 
especially in hands with many actions - was achieved in the 


experiments. The following paragraphs provide highlights of 





the results, with the full results given in Appendix A. 





1. Experiments with Four HMMs 





The HMM with eight states that used 100 centroids 





performed the best. The tables in Section 1 display the 





results of this HMM. As stated in the experimental design, 





the HMM made a prediction based on the first action ina 
hand, then the first two actions in a hand, then the first 
three actions in a hand, and so on, until the end of the 
hand. The results for all predictions are given in Table l. 


Although the accuracy is around 50%, the scores are 








Significantly above baseline for all categories xcept 


folds. 


All Actions - 1880 Predict ions 
ccuracy: 0.5122 | | | 
PT Fold | Low | Med [High | 
Recall 
Precision 


0.6496 | 0.1746 | 0.2192 | 0.5298 
0.7110 | 0.1143 | 0.1862 | 0.4457 
453% 





Table 1. Results for 8-state, 100-centroid four HMM 
experiment for all predictions. 
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It should be expected that with more information 





available, the HMM would perform better. In order to test 


this hypothesis, the performance at certain points in each 





hand is analyzed. The prediction based on the first action 


in a hand can be expected to be low, as there is very little 





information. However, the accuracy of the first prediction 


is 55% (See Table 2), which is better than the overall 





accuracy. The HMM never makes a “low” prediction based on 
the first action. This is not out of the ordinary, as low 
hands can easily be confused with fold hands. In fact, of 








the 27 low hands, 24 were predicted as fold hands based only 


on the first action. 


First Action - 500 Predictions 
Accuracy, 05520 | = || i 
| Fold | Low | Med | High 
Recall 0.2885 


Precision 0.3571 
F-score 0.3192 
Baseline 0.3444 





Table 2. Results for 8-state, 100-centroid four HMM 
experiment for the first prediction in each hand. 





As play continues in a hand, a player will have more 





actions to use in order to judge the strength of an 





opponent’s hand. We hypothesized that using the first three 


actions of a hand to make a prediction should improve the 





performance of the HMM. However, Table 3 shows that the 
accuracy drops considerably. The performance on fold hands 
is extremely low and many medium hands are mistakenly 
labeled as high hands. Note that if the opponent does not 
perform three actions in the hand, the hand is not included 


in this table. The third action of a hand is likely to be 
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just after the flop where the strength of a hand changes 
considerably. This may explain why the performance drops at 


this point in the hand. 


Third Action - 264 Predictions 
Accuracy, 0.3908 | | SC 
| Fold | Low | Med | High 
Recall 0.6539 


Precision| 0.5333 | 0.2200 | 0.1861 | 0.5763 
F-seore 0.6126 
Baseline 0.5361 
465% 





Table 3. Results for 8-state, 100-centroid four HMM 
experiment for the first three actions. 


The sixth action will typically be well after the flop 





when hand strength is relatively stable. Accordingly, the 











performance of the HMM increases significantly over the 





performance based on the first three actions, (See Table 4). 
Again, if the hand does not contain six actions, the 
performance of the hand is not included in this table. Note 


that the precision of folds is approaching 90% while the 








precision of high hand is almost 85% at this point. This 
tells a player that if the HMM predicts a fold, it is 90% 


sure the opponent will fold, and if the HMM predicts high, 
it is 85% sure the opponent has a high hand. Being able to 
distinguish between high and fold at this stage in the hand 


is very important because there is likely to a large pot at 





stake. Making this distinction can earn a good deal of 





money or prevent the loss of more money. Furthermore, all 
of the medium hands that are mislabeled are called high 
hands and most of the mislabeled high hands are called 
medium hands. This indicates the predictions are close and 


perhaps changing the threshold between medium and high hands 





may improve the performance significantly. 
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6th Action - 53 Predictions 
Accuracy 0.6038 | = || i 
| Fold | Low | Med | High 
Recall 0.6471 


Precision| 0.9889 | 0.0000 | 0.1333 | 0.9462 
F-seore 0.73353 
Baseline 0.7816 
B% 


Table 4. Results for 8-state, 100-centroid four HMM 
experiment for the first six actions. 








Although there are only six hands that contain eight or 


more actions, Table 5 shows that a high precision is 





attainable in the fold and high categories. 


ath Action - 6 Predictions 
Accuracy 0.3333 [ =| 
High 
Recall 0.2000 


Precision] 1.0000 | 0.0000 | 0.0000 | 1.0000 
F-score 0.3333 
Baseline 0.9091 





Table 5. Results for the 8-state, 100-centroid four HMM 
experiment for the first eight actions. 





Table 6 shows the results of only the last prediction 





of each hand. The last prediction of each hand uses all the 





actions in that hand -- be it two actions or eight actions - 
- to make a prediction. This table shows the highest 
accuracy for this HMM and a very high precision on fold 


Bang 


hands. This is somewhat misleading because the fold action 








is part of the action vector and is always the last action 





in a fold hand. The fact that the F-score is not higher 
shows that the actions preceding the fold mathematically 


outweigh the fold action in many of the hands. 
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Last Action - 500 Predictions 
Accuracy: O.6400 | =| i 
| Fold | Low | Med | High 
Recall 0.6154 


Precision| 0.9352 | 0.3913 | 0.2000 | 0.4267 
F-seore 0.5039 
Baseline 0.3444 
160% | +46% 





Table 6. Results for 8-state, 100-centroid four HMM 
experiment for the last prediction. 





For thes xperiments, accuracy between 55% and 60% is 
common, with the accuracy generally increasing as the number 
of actions in the hand increases. Additionally, as the 
number of actions increases, the precision of the fold hands 


and high hands increases. 


2. Experiments with Three HMMs 


The HMM with four states and 50 centroids performed 
reasonable well and was consistently between 51% and 55% on 
accuracy. However, the results for the HMM with eight 


states and 100 centroids preformed the better in key areas 





described below. 


Similar to the previous experiments, this method 


achieved an accuracy of 53% on all predictions. Low 





performs 19% better than the baseline score. Most of the 





mistakes in the high and medium categories are in the 





opposite category, again showing that a change in the 








threshold between these two categories may cause significant 


improvements. These results are shown in Table 7. 
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All Actions - 843 Predictions 





Table 7. Results of 8-state, 100-centroid three HMM 
experiment for all predictions. 








This time, as should be expected, the prediction based 











on only the first action is worse than the overall accuracy, 
(See Table 8. Similarly to the first action in the four HMM 


experiment, this model does not predict a low hand based on 





the first action. 


First Action - 175 Predictions 
Accuracy: 





Table 8. Results of 8-state, 100-centroid three HMM 
experiment for the first prediction. 


The performance based on the first three actions is 


considerably higher—exceeding 58% (see Table 9). 





Furthermore, the recall and precision scores are higher in 


all categories here than those recorded in the four HMM 





experiment. 
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Third Action - 175 Predictions 





Table 9. Results of 8-state, 100-centroid three HMM 
experiment for the third prediction. 


The performance of the last prediction is right at the 





average for the three HMM experiments and performed much 


worse than the four HMM experiments (See Table 10). This is 





likely due to the fold data that is inherent in the last 





action of a fold hand, as discussed in the previous section. 


Last Action - 175 Predictions 
ACCuUracy: 





Table 10. Results of 8-state, 100-centroid three HMM 
experiment for the last prediction. 














Except for predictions based on the first three 





actions, this method did not perform better than the four 


HMM experiment. 





3. Experiments with Two HMMs 
Accuracy is much improved in thes xperiments - 
exceeding 85% in some cases. This shows that given broader 


categories, we can improve our performance. 
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Similar to the above experiments, 100 centroids result 
in the highest accuracy. The accuracy for fold hands is 


about 67% based on all actions (See Table 11). 


All Actions - Fold or Not - 1880 Predictions 


Ac curac 0.6718 ae Teal 
| Positive | 


Negative Positive 
Recall 0.6868 
Precision 0.6212 
0.6524 
0.6192 
45% 





Table 11. Results for the 100-centroid fold or not-fold HMM 
for predictions based on all actions. 





Low hands scored the lowest accuracy on the predictions 





based on the first actions and the highest accuracy in the 





last predictions. Table 12 shows that the first action is 
only able to discriminate low or not-low at a 39% rate. As 








expected, this is difficult to determine base solely on the 





first action of a hand. 


First Action - Low or Not - 500 Predictions 

Accurac 03900 | | 
Negative 
0.3679 
Precision 0.9667 


0.6329 0.1210 
0.9723 0.1025 
~45% 





Table 12. Results for the 100-centroid HMM predictions for 
Low or Not-Low based on the first action. 


Table 13 shows that as the hand progresses, it becomes 


easier distinguish low from not-low. In fact, this is where 








the highest accuracy is attained-exceeding 84%. 
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Last Action - Low or Not - 500 Predictions 

Ac curac 0.6460 a 
Negative 
Recall 0.8710 


Precision 0.9626 
0.9145 
0.9723 
-B% 


Table 13. Results for 100-centroid HMM for predictions of 
Low or Not-Low based on the Last Action. 














Interestingly, Tables 14 and 15 show that medium and 





high hands are relatively easy to discriminate on the first 
action. For medium or not-medium hands, accuracy over 70% 


was attained. 


irst Action - Medium or Not - 500 Predictions 
Ac curac 07040 [| | 
Negative 
Recall 0.7259 
Precision 0.9350 
0.8173 
0.9540 
-14% 


Table 14. Results for the 100-centroid HMM for predictions 
of Medium or Not-Medium based on the First Action. 








When discriminating between high and not high, accuracy 








over 66% was attainable on the first action. 


First Action - High or Not - 500 Predictions 

Accurac 06620 [| | 

Negative 
Recall 0.7096 
Precision 0.83868 
0.7688 
0.8839 

-13% 


Table 15. Results for the 100-centroid HMM for predictions 
of High or Not-High based on the First Action. 
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Table 16 shows the best accuracy in all of the 
experiments described in this thesis. As with the 100- 
centroid HMM, the 250-centroid HMM performed best when 
determining low or not-low based on the last action of the 


hand. The accuracy here was over 85%. 


Last Action - Low or Not - 500 Predictions 


Accurac 08560 [| | 
| Positive | 


Negative iti 


Positive 
0.8774 
Precision 0.9674 0.1831 


0.9202 0.2653 
0.9723 0.1025 
5% +159% 





Table 16. 250-centroid HMM for Low or Not Low predictions 
based on the last action. 


B. SUMMARY 


In general, our experiments were successful in the 





following areas. Precision increased significantly as 
increasing numbers of actions are made in a_ hand, 


specifically in fold and high hands. Most high hands that 





were mislabeled were called medium, and vice versa. This 
indicates that adjusting the threshold between these hand 


categories will improve performance. 
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V. CONCLUSIONS AND FUTURE WORK 


A. SUMMARY 


A new method for adversary modeling was explored in 





this thesis. There have been numerous experiments conducted 


on adversary modeling in a wide array of domains—to include 





poker—but none have used Hidden Markov Models in the manner 


described here. This thesis uses Hidden Markov Models to 





predict what an opponent thinks about the strength of his 





hand. First, data was collected from an online corpus and 


mined for the information about the hands of several 





individual players. Next, we choose 13 dimensions of the 





game of poker of which an opponent could use to judge the 


strength of his hand. These game contexts were clustered 





together using the k-means algorithm and then used to train 


Hidden Markov Models. Several models were used to determine 











the most likely model to produce a given sequence of a hand, 





i.e., predict the strength of the hand. Finally, precision, 


recall, and F-scores were used to evaluate the performance 








of the models. The methods in this thesis did not produce 
accuracy above 85% and was usually lower than 60%; however, 
most results were above the baseline, which means’ the 
predictions were better than random. Furthermore, late in 
hands the HMMs were able to make clear distinctions between 
fold hands and high hands—a distinction that will earn a 


large amount of money in the long run. 
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B. FUTURE WORK 


1. Adjusting Hand Strength Thresholds for Hand 
Categories. 


In addition to the work described above, other 
experiments were conducted using different thresholds for 
high, medium, and low hands. Additionally, more hands were 
used in the experiments, resulting in more hands with up to 
eight actions. In one set of experiments, the threshold for 
high hands was raised to 0.90 and the threshold for medium 


hands was raised to 0.70. In another set of experiments, 





the threshold for high was set to 0.85 and the threshold for 





medium hands was set to 0.65. In thes xperiments, there 
were at least 26 hands of at least eight actions; as opposed 


to the six hands with at least eight actions described in 





Chapter IV. Additionally, the distributions of hands in the 





high, medium, and low categories were evenly distributed in 


these new experiments. The predictions based on the first 





eight actions produced many high scores. All predictions 





were well above baseline. For fold hands, the F-score was 








94%, with a recall of 100% and a precision of 89%. The 
precision from high hands was also 100% and the overall 
accuracy score was 69%. This indicates that adjusting the 


thresholds further could result in even better performances. 





Unfortunately, different thresholds might produce different 











results for each opponent - negating one of the greatest 


benefits of using HMMs. 


Zs Modeling Advanced Play in Poker 
Misinformation is inherent in the game of poker. Many 
advanced players will “slow-play” some hands - the technique 
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of playing a very strong hand weakly in order to extract 
more money from your opponent. The opposite of slow-playing 
is bluffing - playing a weak hand as if it were very strong 
in hopes of making your opponent fold. Another advanced 
technique is drawing to a strong hand - where a player who 
does not currently have a strong hand but can call or raise 
because of a high likelihood of getting a strong hand with 


future board cards. 


Modeling these types of hands is extremely difficult. 





Some of the bluff and draw hands could end up in the fold 
category - if the opponent re-raises and then the bluffer 


fold, or if the drawing hand does not catch the draw and 





folds. Despite the difficulties, some data mining 
techniques could be used to classify hands into these 
categories. Then, these hands could be used to train and 


test more HMMs. Futur xperiments would involve high, 





medium, low, blurteé; slow-play, draw, and fold hand 


categories with a corresponding HMM for each category. 





os Principle Components Analysis 
In thes xperiments, the integer labels for the 
centroids were used instead of the centroids themselves. If 





the data point of the centroid contains valuable 
information, using the point instead of label for the point 


may improve the performance. Principle Components Analysis 





(PCA) is a technique used to analyze multidimensional data. 


PCA uses linear combinations of the original dimensions to 





convert the data into a coordinate system. The dimension 





with the greatest variance is the first coordinate and is 





called the first principle component, the dimension with the 


second greatest variance is the second coordinate and is 
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called the second principle component, an so on. PCA can 
also be used to reduce the number of dimensions by ignoring 
the dimensions with less variance. Performing PCA on the 


data could improve the results. 


4. Dimension of Game Context 


Using PCA could also provide insight that can be used 


to choose other dimensions that can be used. For example, 





the Boolean used in this thesis tracks whether or not there 


is a face card on the board. A Boolean for tracking the 





presence of an Ace and another that tracks the presence of a 


King could prove to be more useful. Also, a different 





technique for analyzing the board cards could be used. The 
board strength, probability of straight, probability of 
flush, probability of straight draw and probability of a 





flush draw dimensions used in this thesis could oversimplify 





the threats that a board presents to players. 


Cc. CONCLUSIONS 


Modeling modern adversaries is difficult because of the 








many, differing complexities on small terrorist groups. In 
order to b ffective, one common system for modeling every 
group is necessary. This thesis attempts to create an 


adversary modeling system that is useful in the domain of 


Texas Hold’em Poker becaus of its structure, rules, and 





parallel with wartime adversarial situations. The results 











show that although the accuracy is not sufficient to return 
to the more complex domain of warfare, the Hidden Markov 


Models do perform significantly better than random guessing. 





With more modifications, the accuracy should improve enough 


to conduct experiments with terrorist models. 
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APPENDIX: RESULTS OF HMM EXPERIMENTS 


A. EXPERIMENTS WITH FOUR HMMS 





The first table applies to all of the other tables in 
Section A. It shows the number of predictions made for each 


group of actions. 


7th Action 
| Sth Action [| == Bs 





Table 17. Number of Predictions in each Action Category. 
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0.4720 0.0000 | 0.1789] 0.3200 
| Baseline | 0.7110] 0.1143 | 0.1862 | 0.4437 | Baseline | 0.7879] 0.1025 10.1618 | 0.3444 
-100% | -100% 
eae eee eee eee ee eee, eee ees eee eee 


ard Action Sth Action 


| Fold | Low | Med | High | 
0.701 
0.544 
0.6135 
0.5361 
+14% 
Sa ae aay 
6th Action 


#th Action 
| 
| Low | Med | High | 
Rec all 0.0000 
recision 
F-score | 0. 
Baseline | 0.3636 
-10% |+Baseline| +136% 


0 
oO 
Cc 
= 
a 
o 


vu 


0.6136 | 
0.5361 | 
| +14% | 
2 
ae 


oo 
ma 100 
oo; ja 
=|}; 
m~ jo 


Sth Action 


on) 
ojo|— 
o/ol? 
Slol|z 
oyo|jo|o 


F-score 0.000 
2.000 





Table 18. Results for the 50-centroid, 4-state HMMs. 
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All Actions First Action 
Accuracy: 0.1720 aes Vee 


5 
0.4437 0.7879 
Re ee es QO 
3rd Action Sth Action 
Accuracy: O.3656] | Accuracy: 8.4956 TTT 
| CSS) | Low [| Med | High | | Fold [| Low | Med [ High | 
/Precision| 0.8571 | 0.1809 | 0.1867 | 0.5926 [Precision| 1.0000 | 0.0000 | 0.1429 | 0.7778 | 
a ee es es es 
6th Action 
Accuracy: 0.6660] | | Accuracy: 0.3333] | | | 


Med | High 
| 1.0000 | 0.6000 


3[0. 
a hea) 
Accuracy: O.6667[ [| T  Accuracy: O.5980T | 
| Fold | | Fold | Low | Med | High _| 


0.1025 
+266 % 


Table 19. Results for the 50-centroid, 8-state HMMs. 
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Accuracy[O4734| | |  [Accuracy]osa00[ | | _ 
| Fold | tow | Med | Migh |_| Fo | Low | Med | High 
AS RE SAY (RRSEN (S (SA ES (ERT 
Accuracy|o4014| |_| [Accuracy]oa7a] [| _ 
P| Fold | tow | Med | migh | | Fold | Low | Med | High _ 
Precision] 0.6875 | 0.2000 [0.1892] 0.5762 | Precision] 0.7727 | 0.0000 [0.1639] 07674 
ee ee i a i ee 
Accuacy[uss72| |_| ecwary ose] |] 
P| Fold | Low | Med | High | | ‘Fold | Low | Med | High 
rt tf sf ft | | | 4+f|f| Jf —_ 
Accurecy[Us333| || ccuracy[oerao] «| |] 
P| Fold | Low | Med | High | | Fold | Low | Med | High 





0.9091 | Baseline [0.7879 | 0.1025 [0.1618 | 0.3444 
+Baseline| +250% | -100% | -100% +1% | +220% | +64% | +33% 


Table 20. Results for 75-centroid, 4-state HMMs. 
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Accuracy:|O.3957[ | a eae eee 
PT Fold | PT Fold | Low | Med | High | 
| a ae (a ee es a ee ee ee 
[aed [ees Ce as) ere Aen 
PF | | Fold [ Low | Med [ High | 
[Precision| 0.6957 | [Precision 0.7619 | 
a ee es es ee es es 
en ee ee ee 
| Fool | || Fold | Low [| Med | High | 
ae | ead [ae] [Pees (Se (| (femvemreeeen)| (ey | | ey (| 
eee el ieee ee 
| Fold | | | Fold | Low | Med | High | 
| Recall | 1.0000 | 0.0000 | 0.0000 | 0.4000 | 0.6523 
0.9507 
0.4788 
+Baseline| +250% -2% 


Table 21. Results for 75-centroid, 8-state HMMs. 
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All Actions First Action 


0.3860 


re) eae) iene] (Lanes) aca, ieee maa (pee) ee) eee 
03555] | | —__ Accuracy: [O54e7[ [| [| 
|| Fold | Low | Med [ High [| | Fold [ Low [ Med | High | 
Recall 
Precision| 0.5185 | 0.1600 [0.1942 | 0.5755 [Precision| 0.7500 [0.0769 [0.1936 | 0.9222 
ae eae ee ee a es es a ee ee 
Accuracy:|O.6225.  [ [| ——s Accuracy:]o sooo]. [ [| 
|| Fold | Low | Med | High [| [| Fold | Low | Med | High | 
| F-score [0.7273 | 0.0000 [0.3158] 0.7459] F-score | 0.7500] 0.0000 | 0.0000] 0.6000 
es es CO 
Accuracy|O.5000] | | Accuracy Oe240] [Tt 
|| Sold | Low | Med [ High | | Fold [| Low | Med | High | 
| Recall [1.0000] 0.0000 [0.0000] 0.40007 Reeall [0.7015] 0.2963 [0.2727 | 0.6154 
0.5020 

[Baseline | 0.7879 | 0.1025 [0.1618 | 0.3.44 

% +179% | +26% | 45% 


Table 22. Results for 100-centroid, 4-state HMMs. 
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Accuracy: O0.5122{ | | Fae ees een 
a ee | | Fold | Low | Med | High | 
Sg) RR) (REED (Ces) (a eT SS (| 
aieaeees [ieeereeeh [Seer] ieee (Cee! SomreeT] 
P| Fold PT Fold [Low | Med | High _| 
[Precision| 0.6333 | Precision| 0.7083 | 
a a Se 
es ae i ae a 
ess tS Pold | | | Fold | Low | Med | High | 
he ee he) 
seer ets Epes ees ae 
| Fold | | Fold | Low | Med | High _| 





0.9091 
+Baseline| +250% | -100% | -100% +3% 


Table 23. Results for 100-centroid, 8-state HMMs. 
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Accuracy:|0.3803[ |] ee eens ee 
| Fold P| Fold | Low [| Med | High | 
0.5556 | 0.4318 | 0.0866 | 
a a a (a (ee (ee (Pee) ee (| ee 
[ee (ee (| ae el ee 
| sold | | Fold | Low [| Med | High | 
[Precision| 0.7368 | [Precision| 0.6061 | 
ss 
en ee ee a aes eee 
| Fool | P| Fold | Low [| Med | High | 
aaa (eee [ee] (| (en (een Ay 
ae ee eet ee 
P| Fold | P| Fold | Low | Med | High | 
| Recall | 1.0000) 
0.4 462 

+Baseline| +250 % +179% +30% 


Table 24. Results for 175-centroid, 4-state HMMs. 
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Accuracy:|0.40327 |] ae aes eee 
| Fold P| Fold | Low [| Med | High | 
0.5656 | 0.3409 | 0.2115 | 
a) | (eee eee 
eS ei eS I ee Gee 
| sold | | Fold | Low [| Med | High | 
[Precision| 0.6400 | [Precision] 0.5405 | 
es es 
ae ee (ee a ee 
| Fool | P| Fold | Low [| Med | High | 
ae Te [ee (ee | [ee (ee ey (a 
ae ee ae a 
PT Fold | P| Fold | Low | Med | High | 
| Recall | 1.0000) 
0.4472 

+Baseline| +133% +140% +30% 


Table 2 
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Results for 175-centroid, 8-state HMMs. 
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Accuracy:}O.4112, | oe ees eee 
PT Fold | PT Fold | Low | Med | High | 
a | (a) | 
eed lee Stee al ae a ca 
PT sold | | Fold [ Low | Med [ High | 
[Precision| 0.4376 | [Precision| 0.5500 | 
a a es es ee es es es Gs 
Po ae Sl | ee ee 
| Fool | || Fold | Low [| Med | High | 
ae AE [a] | PA) (eer ne) meee (Perens | || 
eee el pease SS 
P| Fold | P| Fold | Low | Med | High | 
| Recall | 1.0000 | 
0.4000 

+Baseline| +250% +H16% 


Table 26. Results for 250-centroid, 4-state HMMs. 
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Accuracy:|O.39745[ | ae a ieee 
PT Fold | PT Fold | Low | Med | High | 
Recall 
a | (| | || ee a | ee | 
(nel (eee) (ae ae er ee 
PF | | Fold [ Low | Med [ High | 
[Precision| 0.5306 | [Precision| 0.4407 | 
a ee ee es ee es es es Ge 
a ee ee ee ee 
| Fool | P| Fold | Low [| Med | High | 
ae [aay ae] | Pa) (ee) (femme (SPs | | ey (| 
Peel ee eines as 
P| Fold | P| Fold | Low | Med | High | 
| Recall | 1.0000 | 0.0000 | 0.0000 | 0.2000 | 0.7385 
0.7524 

+Baseline| +17 % 


Table 2 





~ 


Results for 500-centroid, 8 state HMMs. 
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B. EXPERIMENTS WITH THREE HMMS 





The first table applies to all of the other tables in 
Section B. It shows the number of predictions made for each 


group of actions. 


Table 28. Number of Predictions in each Action Category. 
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Accuracy 1 085021 Accuracy Accuracy:|0.5429{ | 
Low Ltow | Med | High |__| bow | led | bia 
[Precision | 0.2197 | 0.2690 | 0.6889 | 
aS (EE ES) EE) SED a 
Acouracy:} 0.5402 |__| _}Accuracy: Accuracy:|0.5185| | 
PT Low | Med | High TO | bow | Med | High | 

| Precision | 0.2581 | 0.2188 | 0.7117 | Precision | 0.0500 | 0.2353 | 0.8409 
[ace eee earn ne eee ies Fae ees 
FAccuracy:|0.5366| | Accuracy: [0.3571] | 
| | Low [| Med | High [| Low | Med | High | 
| Precision | 0.1000 | 0.2000 | 0.9048 | 
[ea (reat CAR [SSR (Ay (eee eee 
[Accuracy | 0.6000[ [Accuracy [oss7i[ 
| | Low | Med | High [| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.6000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 29. Results for 50-Centroid, 4-state HMMs. 
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AccuneyTusse| —[|accureeyfusrrit _[ 
a [tow | wad {Aik {| Low | Mes_| Hi | 
[Precision | 0.2661 | 0.2523 | 0.7600, 
ale a es nd 
ccumcy [Osa] —{—|Avowacy uses | ——T 
P| tow | Med [High | | Low | Med | High _ 
Precision | 0.4048 | 0.3044 | 0.7442 | Precision | 0.0000 [0.1724 | 0.8537, 
Ree ce We ee 
Accuracy [US610| | JAccuracy [0.2143] | __ 
P| Low | Med | High | | _Low | Med | High 
[Precision | 0.4000 [0.1765 | 0.9474 | 
Bo i 
Accuracy [ue000| [| JAccuracy[u.s086| | 
P| Low | Med | High | | Low | Med | High 
Recall | 0.0000 [0.0000 | 0.6000, 
[Precision | 0.0000 [0.0000 | 1.0000 


Table 30. Results for 50-centroid, 8-state HMMs. 
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Accuracy | 04567 |} _Accuracy Accuracy:|0.2629{ | 
Low ptow | Med | High |__| cow | ed | bia 
[Precision | 0.2760 | 0.2731 | 0.7616 | 
aS (AER (A EE) ES ee ee 
Acouracy:} 0.5460 |____|__} Accuracy: Accuracy:|O.4691| | 
PT Low | Med | High [| bow | Med | High | 

| Precision | 0.5185 | 0.2830 | 0.7021 | Precision | 0.0500 | 0.1818 | 0.8462 
[asm (eae | (eae i (amen panne iets Pes ee 
fAccuracy:| 0.5610] | Accuracy: [0.2143] | 
| | Low [| Med | High [| Low | Med | High | 
| Precision | 0.1429 | 0.2308 | 0.9048 | 
Seay (Paar ARRAA [ SRA Ry (PN eee eae 
fAccuracy:[0.2000[ J Accuracy [oao7i[ 
| | Low | Med | High [| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.2000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 31. Results for the 75-centroid, 4-state HMMs. 
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Accuracy | 045% | __}__ Accuracy Accuracy:|0.3429{ | 
Low ptow {Med | High} —___ ce | ed | bat 
[Precision | 0.2806 | 0.2804 | 0.7680 | 
aS (SE (RS a ES ee ee 
Acouracy:} 0.5675 |____|_} Accuracy: Accuracy:|0.5185| | 
PT Low | Med | High TO | bow | Med | High | 

| Precision | 0.5000 | 0.2766 | 0.7113 | Precision | 0.0556 | 0.2000 | 0.8605 | 
[aa (Sa (ema (ea (iene ees ey as 
[Accuracy:|0.5864| | Accuracy: [0.4286] | 
| | Low | Med | High | | Low | Med | High | 
| Precision | 0.1250 | 0.2727 | 0.9091 | 
i ee 
[Accuracy [0.4000[ [Accuracy [0.5086[ 
| | Low | Med | High [| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.4000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 32. Results for the 75-centroid, 8-state HMMs. 
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Accuracy | 04781 Accuracy Accuracy:|0.2971{ | 
Low L tow | Med | Hi | ——_—_ cow | ed | bia 
[Precision | 0.2766 | 0.2836 | 0.7608 | 
aS (SE aS) RE) ES EE 
Accuracy} 0.5287 |____|_} Accuracy Accuracy:|0.6432| | 
PT Low | Med | High TO | bow | Med | High | 

| Precision | 0.4000 | 0.2800 | 0.7191 | Precision | 0.0769 | 0.2308 | 0.8810 
Face (en re ea (cme panes eae enc si 
fAccuracy:| 0.6098] | Accuracy: [0.4286] | 
| | Low [| Med | High [| Low | Med | High | 
| Precision | 0.0000 | 0.2500 | 0.6800 | 
[ea (ete (seer (Sa (Ry (isa ieee aaa) 
[Accuracy [0.4000[ | Accuracy: [o4es7[ | 
| | Low | Med | High [| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.4000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table: 3:3:; Results for 100-centroid, 4-state HMMs. 
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Accuracy 1 08286 1 Accuracy Accuracy:|0.4743{ | 
Low L bow | Med | Hig} —_ bow led | bia 
[Precision | 0.2863 | 0.2731 | 0.7310 | 
ES (ESE) (ERS) Nee | ET 
Accuracy} 0.5605 |____|__} Accuracy: Accuracy:|0.6062| | 
PT Low | Med | High [| bow | Med | High | 

| Precision | 0.5217 | 0.3269 | 0.7273 | Precision | 0.0000 | 0.2000 | 0.8571 | 
[acme (ame (ein (ee ena ea Pe 
fAccuracy:|0.5864/ | Accuracy: [0.5000] | 
| | Low [| Med | High [| Low | Med | High | 
| Precision | 0.0000 | 0.1667 | 0.6462 | 
[aR (area (a [ea] Ry (EA eee ee 
[Accuracy [0.2000[ [Accuracy [0.5086[ | 
| | Low | Med | High | {| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.2000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 34. Results for the 100-centroid, 8-state HMMs. 
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Accuracy 1 04696 | _}__ Accuracy Accuracy:|0.2971{ | 
Low _Low | Med | High | —_ cow | ed | bia 
[Precision | 0.2437 | 0.2923 | 0.7788 | 
SS (SE RRS) (eee) ES a 
Acouracy.} 056/5 |____|_} Accuracy: Accuracy:|O.4321| | 
PT Low | Med | High TO [bow | Med | High | 

| Precision | 0.3667 | 0.3214 | 0.7727 | Precision | 0.0500 | 0.1739 | 0.7895 | 
[asm (seh area Pee een ieee Pat eee 
fAccuracy:|0.5122| | Accuracy: [0.2143] | 
| | Low [| Med | High | | Low | Med | High | 
| Precision | 0.0909 | 0.2727 | 0.8947 | 
[ent eae (ei [ay (Ay (ee eee ae 
[Accuracy [0.4000[ [Accuracy [o4e00T 
| | Low | Med | High [ {| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.4000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 35. Results for the 175-centroids, 4-state HMMs. 
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Accuracy | 04082 1 Accuracy Accuracy:|0.3486{ | 
Low ptow | Med | Hig |__| bow _| led | bias 
[Precision | 0.2686 | 0.3064 | 0.732 | 
aS (EE (SS) RE) eS a ee 
Acouracy:} 0.5632 | ____|_} Accuracy Accuracy:|0.6062| | 
PT Low | Med | High TO | bow | Med | High | 

| Precision | 0.4375 | 0.3148 | 0.7614 | Precision | 0.0588 | 0.2000 | 0.8182 | 
[ase aaa (en (ena meee eee Pes aes 
fAccuracy:|0.5864| | Accuracy: [0.3571] | 
| | Low [| Med | High [| Low | Med | High | 
| Precision | 0.1111 | 0.2500 | 0.8750 | 
[ea ree (iene (PS Sy (Ay (sane ee Py) 
[Accuracy [0.4000[ [Accuracy [o4ea6[ | 
| | Low | Med | High [ {| Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.4000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 36. Results for the 175-centroid, 8-state HMMs. 
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Accuracy | 04004 | __}__ Accuracy Accuracy:|0.3371{ | 
Low Liew | Med | High |__| cow led | fia 
[Precision | 0.2509 | 0.3288 | 0.7600 | 
aS (SR aS SE | SES Ee ee 
Acouracy.} 0.6920 |____|_} Accuracy Accuracy:|O.4691| | 
PT Low | Med | High [| bow | Med | High | 

| Precision | 0.3721 | 0.3667 | 0.7525 | Precision | 0.0800 | 0.2857 | 0.8571 | 
[aa (aan ea (mane pene ee ee es 
fAccuracy:|0.4146| | FAccuracy:| 0.0000] | 
| | Low [| Med | High | | Low | Med | High | 
| Precision | 0.0833 | 0.1667 | 0.6235 | 
eae (Sian (Sa (a (ay (A eee 
fAccuracy:[0.2000[ [Accuracy [o44s7[ | 
| | Low | Med | High | | Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.2000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 37. Results for the 250-centroid, 8-state HMMs. 
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Accuracy | 040s | _ Accuracy Accuracy:|0.3371{ | 
Low p tow | Med | High |__| bev | ed | bia 
[Precision | 0.1962 | 0.2322 | 0.7373 | 
a fi 
Accuracy} 05067 |__| _} Accuracy: Accuracy:|O.3210| | 
PT Low | Med | High TO | bow | Med | High | 

| Precision | 0.2766 | 0.2619 | 0.7629 | Precision | 0.0556 | 0.1876 | 0.7241 | 
[acme eae (ear ey eee ae Eee a 
fAccuracy:|0.4146| | Accuracy: [0.3571] | 
| | Low [| Med | High | | Low | Med | High | 
| Precision | 0.0526 | 0.2500 | 0.8333 | 
Seay (SP (am [aaa Ae (eee ieee ey) 
fAccuracy:[0.6000[ [Accuracy [0.31437 
| | Low | Med | High | | Low | Med | High | 
| Recall | 0.0000 | 0.0000 | 0.6000 | 
| Precision | 0.0000 | 0.0000 | 1.0000 | 


Table 38. Results for the 500-centroid, 8-state HMMs. 
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Cc. EXPERIMENTS WITH TWO HMMS 





The first table applies to all of the other tables in 


Section C. It shows the number of predictions made for each 
group of actions. All HMMs in Section C contained eight 
states. 





Number of Predictions 
All Actions 1880 


First Action 





Table 39. Number of Predictions in each Action Category. 


Accuracy O.6718 | — [Accuracy 0.6200 


0.7015 

a aes ee eee eee eee 
[Accuracy] 06.6915 | [Accuracy) 0.7600 | 
| | Negative | Positive | | Negative | Positive | 


Table 40. Results for 100-centroid HMMs predicting fold or 
not-fold. 
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All Actions 
0.6496 | J Accuracy] 0.6620 | 
| Negative | Positive | S| Negative | Positive 
0.6064 0.4808 
Precision | 0.8525 | 0.4275 [Precision] 0.9598 | 0.3030 | 
F-score 
Baseline Baseline 
Ss ee ee ee 
-Ascursey | 0.6197 | ——_—_ Accuracy| 0.6520 [| | 


Table 41. Results for 100-centroid HMMs predicting high or 
not-high. 








Accuracy] 0.6117 [| [Accuracy[ O7o40 [| 
[| Negative | Positive | (| Negative | Positive | 
F-score 

Baseline Baseline 
ae Se es eee eas aS 


Third Action 
0.4894 af O67e0 [| 


Table 42. Results for 100-centroid HMMs predicting medium or 
not-medium. 
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All Actions 
0.6032 | J Accuracy| 0.3900 | 
| Negative | Positive | S| Negative | Positive 
0.6053 0.7778 
Precision | 0.9562 | 0.0853 [Precision] 0.9667 | 0.0656 | 
F-score 
Baseline Baseline 
aS ree eee 
-Ascurmey |_0.369_|_——_—_ Accuracy| o.8460 [| | 


Table 43. Results for 100-centroid HMMs predicting low or 
not-low. 








Accuracy] 0.6612 [  [Accuracy[ 06220 [| 
[| Negative | Positive | (| Negative | Positive | 
F-score 

Baseline Baseline 
a 8 a 


Third Action 
0.6021 arte Giana 7700 [| 


Table 44. Results for 250-centroid HMMs predicting fold or 
not-fold. 








71 


All Actions 
06314 | [Accuracy] 0.6640 | 
| Negative | Positive | S| Negative | Positive 
0.6191 0.4712 
Precision | 0.8213 | 0.4095 [Precision] 0.9973 | 0.3025 | 
F-score 
Baseline Baseline 
a ae a a ae ee 
-Ascurmey |_0.5848_|_—_—_ Accuracy| O.6260 | | 


Table 45. Results for 250-centroid HMMs predicting high or 
not-high. 








Accuracy] 0.6261 [ [Accuracy] O7o40 [| 
[| Negative | Positive | (| Negative | Positive | 
F-score 

Baseline Baseline 
ee ae ee eee eee ee 


Third Action 
0.5246 af 07220 [ 


Table 46. Results for 250-centroid HMMs predicting medium or 
not-medium. 
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All Actions 
0.6953 | Accuracy] 0.4980 | 
| Negative | Positive | S| Negative | Positive 
0.6648 0.5185 
Precision | 0.9545 | 0.0892 [Precision] 0.9476 | 0.0556 | 
F-score 
Baseline Baseline 
Sy ee eae 
-Accursey | 0.6998—_| ———_ Accuracy| o.8560 | | 


Table 47. Results for 250-centroid HMMs predicting low or 
not-low. 








Accuracy] 0.6128 [| [Accuracy[ 05640 [| 
[| Negative | Positive | (| Negative | Positive | 
F-score 

Baseline Baseline 
ee ee eet eee eee eae 


Third Action 
0.6338 orate Giana 07090 [| ss 


Table 48. Results for 500-centroid HMMs predicting fold or 
not-fold. 








73 


All Actions 
0.6420 | J Accuracy] 0.6940 | 
| Negative | Positive | S| Negative | Positive 
0.6414 0.3942 
Precision | 0.8186 | 0.4172 [Precision] 0.9293 | 0.3130 | 
F-score 
Baseline Baseline 
Ss eee | ee ee 
-Ascursey | 0.6197 | —_—_ Accuracy| 0.6120 [| | 


Table 49. Results for 500-centroid HMMs predicting high or 
not-high. 








Accuracy] 0.6372 [ [Accuracy[ O6940 [| 
[| Negative | Positive | (| Negative | Positive | 
F-score 

Baseline Baseline 
ae ae a eee ee 


Third Action 
0.5739 anf O6740 [| 


Table 50. Results for 500-centroid HMMs predicting medium or 
not-medium. 
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fAccuracy| 0.6011 | | 
| Negative | Positive | S| Negative | Positive 
0.7407 
[Precision | 0.9647 | 0.0662 | 
F-score 
Baseline Baseline 
ae ee a (ee ae) eee 
-Arcursey |_ 0.6548] | Accuracy | O.7260 | 


Table 51. Results for 500-centroid HMMs predicting low or 
not-low. 
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